DOCUMENT RESUME 



ED 425 331 



CE 077 705 



AUTHOR 

TITLE 

INSTITUTION 

ISBN 

PUB DATE 
NOTE 

AVAILABLE FROM 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Seyfried, Erwin 

Evaluation of Quality Aspects in Vocational Training 
Programmes. Synthesis Report. CEDEFOP Document. 

European Centre for the Development of Vocational Training, 
Thessaloniki (Greece) . 

ISBN- 92 -828-5161-3 

1998-04-00 

51p. 

Bernan Associates, 4611-F Assembly Drive, Lanham, MD 
20706-4391; (Tel): 800-274-4447 (Toll Free); e-mail: 
query@bernan. com; http : //www. bernan . com (catalogue no . 
HX-18-98-437-EN-C: 6 European Currency Units) . 

Reports - Research ( 143 ) 

MF01/PC03 Plus Postage. 

Comparative Analysis; Context Effect; Education Work 
Relationship; Educational Environment; *Educational Quality; 
Evaluation Criteria; Evaluation Methods; Foreign Countries; 
♦Formative Evaluation; Meta Analysis; Models; National 
Surveys; Postsecondary Education; *Research Methodology; 
Secondary Education; Standards; State of the Art Reviews; 
♦Summative Evaluation; Synthesis; *Vocational Education 
♦Europe; Quality Indicators 



ABSTRACT 



National evaluations of quality aspects of vocational 
training programs (VTPs) in five European countries- -Belgium, Germany, 

France, Greece, and Portugal- -were reviewed to determine how VTP quality is 
currently being evaluated and identify a more coherent approach to evaluating 
VTPs. Special attention was paid to how existing evaluation studies consider 
relationships between the contents, kinds, and methods of vocational training 
and trainees' vocational perspectives and to how the needs of trainees in 
different target groups are represented in the design of VTPs. The CIPP 
(context, input, process, product) model of evaluation developed by 
Stufflebeam and associates was used to systematize the methodological 
approaches of the studies examined. In terms of methods, the evaluation 
studies reviewed varied considerably. Hardly any studies examined the links 
between the context, process, and product of VTPs, although many studies 
focused on a single element from the cycle. It was recommended that a 
coherent approach to evaluating VTPs be devised that gives consideration to 
both process-oriented and product-oriented indicators of quality. The need 
for greater convergence between the debates on training policy and labor 
market policy was also emphasized. (Contains 73 references.) (MN) 
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Foreword 



In its project on "Quality in VET', CEDEFOP commissioned the present study, which 
shows how evaluations of quality aspects of vocational training programmes are made. 

It is a study which emanates from and deepens the reflection of the OAED/CEDEFOP 
Conference on the Evaluation of European Training, Employment and Human Resource 
Programmes, held on 30 November and 1 December 1995 in Athens. It is a meta- 
analysis of evaluation studies of VET programmes in five European countries with 
special emphasis on quality. 

Aware that quality is a constant aim and that even the most sophisticated evaluations 
are not perfect, we decided to focus on quality and evaluations and examine their 
interrelations in an effort to draw conclusions and to come forward with suggestions for 
a more coherent approach in the training field. A rather difficult task, especially when it 
refers not only to quality within the training process itself but also to its relations to labour 
market policies. 

Besides, an evaluation is never neutral. Evaluating public (and private) training 
programmes serves different purposes and may have important consequences for all 
involved. But the recent proliferation of training programmes - in an effort to adapt the 
work force to technological and economic change and to combat unemployment - 
prompts us to reflect on their effectiveness and on their possible improvement. In other 
words, evaluation steps in with its many interdisciplinary issues concerning 
employment, vocational training and social policies. 

Based on the experiences of previous evaluations made in Germany, Belgium, Greece, 
France and Portugal as reported in the national studies conducted by the research 
teams mentioned in point 1.3 of this publication. Professor Dr. Erwin Seyfried, who is the 
scientific coordinator of this exercise, was able to structure the many different kinds of 
information provided into a coherent whole and to formulate suggestions for improving 
upon previous research on evaluations. 

We wish to thank him as well as the other research participants for their invaluable work 
of locating and analysing evaluation documents which were often considered 
confidential and even compromising. 



Tina Bertzeletou 
Project coordinator 



Stavros Stavrou 
Deputy Director 



Table of contents 

1. Introduction 1 

1 . 1 Objective of the study 1 

1.2 Methodological approach 2 

1.2.1 Consideration of the needs of the trainees 3 

1.2.2 Consideration of employment perspectives in training programmes 3 

1.2.3 Consideration of quality aspects of employment 4 

1.2.4 Areas of investigation 4 

1.3 Organisation of work 5 

2. Evaluation of vocational training programmes; context, objectives 

and functions 7 

2.1 Different interests - different concepts 7 

2.2 The context for valuation 8 

2.3 Evaluationof quality aspects 9 

3. Models for the evaluation of vocational training programmes (VTP) 1 1 

3.1 Ex-ante evaluation 1 1 

3.2 Process-orientated evaluation 1 2 

3.3 Ex post evaluation 13 

4. Evaluation of vocational training programmes; The state of the art in 

the countries involved 1 5 

4.1 Belgium 1 5 

4.2 Germany 16 

4.3 Greece 1 7 

4.4 France 18 

4.5 Portugal 20 

5. Evaluations of quality aspects of vocational training programmes 21 

5.1 Evaluationof context 21 

5.2 Evaluation of input 22 



5.3 Process-orientated evaluations 23 

5.3.1 The conditions ofthe trainees 24 

5.3.2 The trainers qualifications •25 

5.3.3 Evaluation ofsituative process variables 25 

5.4 Product-orientated evaluation 27 

5.4.1 Assessment of learning achievement 27 

5.4.2 Evaluation ofthe learning transfer 28 

5.4.3 Evaluation of employment prospects 29 

5.4.4 Quality aspects of employment 31 

5.4.5 Longitudinal studies 32 



6. Summary and conclusions: The dynamic relationship between 

process and product 34 



References 



39 



1 Introduction 



The present study is part of CEDEFOP’s work programme on quality issues of 
vocational education and training, which was launched after the corresponding 
Resolution of the European Council in December 1994. 

1.1 Objective of the study 

The objective of the present study was to analyse and to compare in a European frame 
work if and to what degree the evaluation of quality aspects in vocational training 
programmes (VTP)has been taken into consideration by relevant evaluation studies so 
far. In the context of this study, the quality of vocational training programmes is primarily 
seen in relation to the needs of the target groups in question as well as in relation to 
their employment prospects after completion of the training and the quality of the jobs 
obtained. 

National studies were carried out in Belgium, Germany, Greece, France and Portugal. In 
the countries involved different approaches have been developed to tackle this question. 
There is no consistent methodology concerning the evaluation of VTP and the studies 
which have been reviewed have even less in common concerning the evaluation quality 
aspects. Therefore the present synthesis is more a mosaic than a coherent picture, the 
report stresses contradictional developments but nevertheless it is able to summarise 
and to synthesise the findings and to identify some common elements on the European 
level. 

In particular this study deals with the relationships between the contents, kinds and 
methods of vocational training on the one hand and the employment perspectives of the 
trainees on the other. The main questions of the study have been; have such 
relationships been examined so far through evaluation studies carried out in the 
countries included? What are the methodological approaches used to analyse such 
relationships? What can be said about the quality of the methodological approaches 
which had been used in existing evaluations and their possible usefulness in a 
European framework? Which basic data is needed to come to valid statements 
concerning the relation between quality issues in vocational training and employment 
perspectives? In general, the important issue in the framework of this study is dealing 
with the question, if and to what degree the dynamic relationship between vocational 
training programmes and employment is reflected in existing evaluation studies. 

A second field of investigation in this study tackles the question, how the needs of 
trainees belonging to different target groups are represented in the design of vocational 
training programmes. In this context needs are defined in a triple way. A first 
understanding is about the social starting conditions of the target group, which must be 
reflected in the overall design of training programmes in order to reach the envisaged 
target group: does a training programme for single-parent adults, for example, include 
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additional child care support? The importance of such specific requirements for different 
target groups is obvious, but should it also be evaluated in a scientific way? A second 
meaning of needs relates to the question of whether the contents and methods of the 
training courses are in accordance with the previous knowledge of the target groups 
envisaged. The third notion of needs again deals with employment prospects: how far 
does the training programme reflect a realistic perspective concerning employment after 
completion of the training? 



1.2 Methodological approach 

The research undertaken in the framework of this study focused on a close review and a 
critical analysis of evaluation studies which had been available in the countries 
included. The study is thus a kind of meta-analysis of sources dealing with the 
examination of relationships between vocational training programmes and employment. 
According to the following illustration the meta-analysis will concern evaluation studies 
which can be located in the angle between the needs of the trainees, the quality of 
vocational training programmes and the employment prospects of the trainees: 



Table 1: Analysis of evaluation studies 




-targeting of programmes 
-identification of needs 



- kind and type 
of training 
-consideration of 
employment 
perspectives 



- kind and type of 
employment 

- quality of 
employment 



Each national expert had to analyse evaluation studies in his/her country and to proove 
which relationships inside this angle have been examined so far. But not only the 
results of the analysed evaluation studies were interesting in the framework of this 
research project, furthermore the present study was interested in the methodological 
approaches which have been used to analyse quality aspects of vocational training 
programmes and their effects on employment so far in the countries involved. 
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1.2.1 Consideration of the needs of the trainees 

Whereas the relationship between vocational training and employment deals with the 
question if and how far the training industry serves the needs of the labour-markets, this 
subject is about the needs of the participants in training programmes. The central 
question is, how far are the needs of the trainees taken into consideration and which 
methods are used to reflect their starting conditions in the design and in the 
implementation of training courses. The main questions to investigate in this field were: 

■ Which instruments are used in the different countries to ensure that the trainees, 
needs are included in the design and in the practical implementation of vocational 
training programmes? Have these instruments ever been assessed? 

■ In which fields, relevant for the implementation of training according to the specific 
needs of different target groups (for example: quality of trainers; quality of technical 
equipment; relationship between theory, practice and work experience; 
appropriateness of training methods), have assessment studies been carried out? 

■ Are there any evaluation studies which determined the consideration of the needs of 
trainees in training programmes? Did evaluation studies deal with the question of 
whether trainees needs are met by the programmes? 

■ Are there any evaluation studies dealing with the forecasting of employment needs for 
the target groups envisaged? 



1.2.2 Consideration of employment prospects in training programmes 

Employment prospects of trainees can be seen as one of the most crucial quality 
issues of vocational training programmes. That is why the research primarily had to 
analyse how employment prospects of (former) trainees are considered in the design of 
training programmes. The questions which had been examined can be listed as 
followed: 

■ Is there any evaluation of employment prospects by the responsible bodies either on 
national, regional or local levels or through the promoters of vocational training before 
establishing (new) training programmes (ex-ante evaluations)? 

■ Is there any institutionalised cooperation between training bodies, employers, 
organisations and trade unions on national, regional or local level which tend to 
strengthen the links between training and employment? Are there evaluation studies 
examining the effects of the different kinds of cooperation on the quality of training as 
well as on the employment prospects of trainees? 

■ Is there any evaluation of outcome in terms of employment prospects of the (former) 
trainees through the training bodies or other relevant organisations? 
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1.2.3 Consideration of quality aspects of employment 

If analysing the relationship between training programmes and employment the main 
focus of evaluation studies often is on the question whether the trainees got a job or if 
they remained unemployed. The quality aspects of employment often are neglected. The 
present study therefore examined the methods used in each of the countries involved to 
include quality issues of employment when evaluating the outcome of training 
programmes. The main questions analysed were: 

■ Which methods and criteria have been developed so far in evaluation studies on 
employment effects of vocational training programmes, to include quality aspects of 
the employment posts which have been reached by former trainees? 

■ Are there evaluations studies examining the correspondence between the level of 
qualifications obtained by the trainees and the level of qualification afforded by their 
present job? 

■ Are there evaluations considering the type of employment and the type of contract 
(part-time, full-time, short-term, seasonal, time-limited, unlimited, self-employment) of 
former trainees when evaluating the employment effects of training programmes? 

■ Has the type and the importance of payment conditions (combination with subsidies) 
been taken into consideration when evaluating the employment effects of training 
programmes? 

■ What are the time-limits within which the effects between training and employment 
have been examined? Have there been long-term follow-up studies carried out? 



1.2.4 Areas of investigation 

Given the complexity of the subject the area of investigation had to be limited. The 
following priorities were set as regards to target groups and according to the type of 
training. Following the factual availability of evaluation studies in the countries included 
there was, however, some reshaping of the scope of the research project during the 
process of work. 

The study is focused on the analysis of evaluations of vocational training programmes 
for the unemployed, for people threatened by unemployment, for people with low and 
insufficient qualifications, for unskilled target groups and various disadvantaged 
populations. This is because many human resource programmes in favour of these 
target groups are financed through European channels, mainly the European Social 
Fund but also for example the Community Initiatives Employment and ADAPT. The 
evaluation of these programmes is a requirement which comes along with European 
funding and that is why evaluations studies of these programmes are available in each 
of the countries involved. 
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Studies on continuous training programmes have been included because they are 
addressing mainly skilled workers, and thus represent a quite different group with 
different needs to those mentioned previously. The analysis of evaluations of in-house 
training of large companies has been taken into consideration because of the fact that 
such programmes often are closely tailored to the needs of the companies’ population. 
Therefore they can be seen as ideal cases to analyse how the preconditions from the 
trainees’ sides are reflected in the design of training and are evaluated in relevant 
studies. Furthermore in-house training in large companies does have a genuine 
interest in the question of transfer of what has been learned during the training into the 
work situation. Evaluation studies concerning the transfer problem contain information 
about the relations between the contents and methods of training and its effects in the 
employment situation. 



1.3 Organisation of work 

The study was carried out with the same methodological approach in each of the 

countries involved. The research teams consisted of: 

Belgium Prof. Mateo Alaluf 

Esteban Martinez 
Adinda Vanheerswynghels 
Free University of Brussels 

Germany Prof. Dr. Erwin Seyfried 

Research Unit for Vocational Training, 

the Labour-market and Evaluation at FHVR Berlin 

Greece Dr. Dimitris Chassapis 

Aristotelian University Thessaloniki 

France Fabienne Berton 

Prof. Jean-Francois Germe 
CNAM Paris 

Portugal Dr. Emilia Andrade 

Universidade Nova de Lisboa 



* * * 



I would like to thank all these contributors to the synthesis report. Special thanks are 
addressed to Tina Bertzeletou from CEDEFOP who brought the different teams together 
and coordinated the discussions amongst them. 



Berlin, April 1 998 
Erwin Seyfried 
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2. Evaluation of vocational training programmes: context, 
objectives and functions 

Evaluation of training systems, programmes or courses tends to be a demand of a 
social, institutional or international nature (Figari, 1994). The demand of a social nature 
became stronger with the participation in training of social partners and other non- 
traditional actors. Evaluation is also a growing institutional demand, since the decision- 
makers need to understand the processes, but also to control and act upon the effects 
of their guidelines. Evaluation is still an international demand given the Community 
involvement in the financing of training in the ED Member States. The re-negotiating of 
funds, the continuity or institutionalisation of programmes are often grounded or justified 
with results and conclusions of the external or internal evaluations that were made. 

2.1 Different interests - different concepts 

The multi-interest character of all these demands influences the conditions and the 
theoretical framework of evaluation and are a relevant condition of its functions and 
objectives. The functions and objectives of evaluation are structuring points that 
influence every decision related to each evaluation exercise - the data to be collected 
and analysed, the actors to be involved, the methodology to be used or the use to be 
given to results, conclusions and recommendations. Any evaluation model depends on 
and is related to the context in which it is designed and where it is applied, and has the 
purposes that stakeholders decide to evaluation - each evaluate exercise 'obeys' pre- 
defined objectives and builds (implicit or explicitly) its own rationality. Maybe there is a 
danger to avoid: the danger of turning evaluation into a ‘mechanism of information that 
allows us to discover only what was foreseen a priori’ as Planas (1994: 36) stated. 

Authors such as Stufflebeam and Patton recognise another important dimension to 
evaluation - the integration of learning: ‘The evaluation of programmes is done to 
support decisions, clarify options, reduce uncertainty and produce information about the 
programmes and the politics within the contextual boundaries of time, place, values and 
politics’ (Patton, 1989: 14). Stufflebeam stated that ‘the most important purpose of 
evaluation is not to prove, but to improve’ (1988: 151) - which means that if evaluation is 
done to support decisions, then these should be taken according to the improvements 
to be made in the programme. The processes of learning related to the quality of 
training can be stimulated with evaluation. Supporting the actors (both those involved in 
the processes of decision or those are connected to practice) in the diagnosis of 
problems and needs, evaluation can help to conceive and build the 'guidelines', to the 
change and influence the processes of change and innovation. Change that occur 
inside a collective learning process are able to build a new social practice (Crozier and 
Friedberg, 1977). In this way, evaluation can speed up the processes of learning by 
bearing out to what (without it) could be neglected or misunderstood: ‘the value of 
evaluation is not limited to the destiny of a programme to be studied ... since it clarifies 
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the phenomena and so can have a deep influence in the way the problem is understood 
and in the form of future programmes’ (Cronbach, 1980: 139). 

In the Portuguese context, for example, the need to integrate the dimension of 'collective 
learning' in evaluation is stressed, among others, by Rodrigues ‘the function of 
evaluation in Portugal can be much more than a way of controlling efficiency and efficacy 
of the allocation of financial resources, even if this is very important’. Evaluation can be a 
‘support to the capacity of learning and of innovation’ (1993; 7). The effects of evaluation, 
as a learning process, will be produced at different levels; in the production of 
information, in the support to negotiation and to the institutional and cultural change of 
the actors involved in the development of the programme. Stern stresses the 
importance of participative and collaborative evaluation (1992). The level of participation 
of the actors and their capacity for negotiation depends on knowledge, and knowledge 
can be produced in the evaluative process producing enlightenment and empowerment. 
It is this empowerment of actors through knowledge that develops their capacity to 
negotiate and makes them capable of changing their practices. This can happen when 
evaluation is integrated in a process of personal and organisational development. 
Which means that evaluation should be closely related to the decision-making process 
at every level. 

Evaluation should not be understood as an activity that only concerns those who 'order' it 
or those who do it, where the actors of the evaluated object are eventually seen as one 
of the audiences or just a resource of information. There is a strong relationship 
between the participation of the different stakeholders in the evaluation activities and the 
acceptance or recognition of the value and importance of the results. The actors who 
participate in evaluation are in better position to implement any changes suggested by 
the activities and results of evaluation, i.e. of integrating evaluation in the practices in the 
direction of change and in the improvement of training. 

The aims of evaluation, (whose object is not students' knowledge) are mainly developed 
around the following axes: as an instrument to support decisions (Stufflebeam), as a 
process that fosters social learning and supports innovation (Rodrigues, 1992); as a 
process that, involving negotiation and the participation of the social actors, leads to 
their empowerment (Guba, 1990; Stern, 1992). Evaluation is designed and developed 
according to needs and objectives, concerns, interests and values. We adopt the criteria 
that we think are the most suited to the objectives we want to reach. We choose models 
and methods according to the aims to be attained or to the decisions to be taken. 



2.2 The context for ‘valuation’^ 

Those who conduct evaluation studies and those who participate in the training 
programme being evaluated have brought and cannot avoid bringing to the evaluation 
their personal sets of values. The context of valuation comprises the basic values of the 

* The term valuation is used to stress the existence and the appliance of values in the process of evaluation. 
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evaluation, including all the biases, values, attitudes and pressures from which it arises. 
The contexts of valuation is characterised by complexity and variation according to the 
case but may be roughly distinguished into three groups: 

■ contexts of valuation providing an unconditional service to the agencies which have a 
major control over the allocation of training resources. The evaluation studies adopt 
the values of those holding office and offer information which will help them to 
accomplish their policy objectives. These studies essentially provide management 
consultation and their criterion of success is client satisfaction. Their approaches and 
methods must therefore to be credible to the policy-makers and are not laid open to 
public criticism. These evaluation studies are actually owned by the training 
bureaucracy which make use of their results according to its interests. That is why 
these evaluation studies are usually not published or widely accessed; 

■ contexts of valuation providing a conditional service to the agencies which have major 
control over the allocation of training resources. These studies offer external 
validation of policy in exchange for compliance with their recommendations. Their 
values are derived from the perception that the evaluation study groups hold about 
institutional and social obligations of the training bureaucracy. These studies 
essentially provide expert advice. Their approaches and methods must therefore yield 
scientific proofs, because their power reference is the academic research community 
and their reports are usually published in specialised journals; 

■ contexts of valuation providing an information service to anybody concerned about the 
characteristics of a training programme. The evaluation studies adopting such a 
context they respect value pluralism and seek to represent a range of interest in their 
issue formulation. Their basic values are derived from the respect to the right of the 
concerned to be informed. The main evaluative activity adopted is the collection of 
definitions of, and reactions to, the training programme. The criterion of success of 
these evaluation studies is the range of the audience served. 

The outlined contexts of valuation are, of course, ideal types and many evaluation 
studies contain elements of more than one of them. The danger also of using such 
value-laden words to describe them is recognised. However it may to be a useful 
distinction between on the one hand, the evaluation of a training programme by- 
objectives model and on the other hand the goal-free evaluation approaches. 

2.3 Evaluation of quality aspects 

Training can be viewed as a form of social engineering for rectifying deficiencies and 
therefore as a process of producing certain desired changes in the behaviour of 
trainees. On the other hand it can be viewed as a form of personal professional 
development and therefore as a process of change which is self-regulating and 
autonomous and which involves enabling, fostering and providing opportunities for the 
professional support and promotion of trainees. 



The task of evaluation according to the first view is necessarily to assess the 
effectiveness of what has been or is done to the trainees in the light of the changes in 
behaviour which are required to remedy the perceived defects. Evaluation according to 
the second view is the concomitant of the professional development of trainees 
adopting therefore the perspective of the trainees in assessing training activities from 
the standpoint of their training needs and professional rights. The evaluation task is 
mainly concerned with process rather than product criteria since it seeks to assess 
training activities in the light of norms of actions referred to by educational concepts. 

When considering the evaluation of quality of training we have to be aware that quality is 
a multi-dimensional concept. Berghe (1996: 5) tries to organise some of the most 
important viewpoints and perceptions to quality: 

■ quality as excellence, as something special 

■ product-oriented quality (quality can be measured) 

■ quality of as the fulfilment of customer expectations 

■ process-oriented quality (quality is conformance with specifications) 

■ price/benefit-oriented quality (the value approach). 

In its major sense the quality of training should relate to the process of development of 
the trainee, to the process that makes him able to acquire and develop knowledge and 
competencies allowing him to answer the diversity of challenges along life. Aubegny 
(1989) considers that the only criteria of institutional evaluation is the impact that training 
has on the development of the trainees. 

The evaluation of quality demands a systemic approach that helps to analyse deeply the 
relationship between politics, providers of training, social partners and the trainees, 
between their needs, interests and objectives. The approaches to quality tend to 
consider different moments and axes in its construction and analysis (Le Boterf, 1992; 
Dupouey, 1991; Berghe, 1995) - the moment of definition of policies, conception and 
development of the training and the application of competencies, according to Le Boterf 
or the quality of design and the quality of conformance, according to Berghe. The quality 
of training should be evaluated according to national/general but also according to local 
criteria. The general/national criteria would allow the comparability between training 
projects that emerged from the same guidelines, but on the other side the local criteria 
would allow the analysis of the training project in its own context. So training would be 
seen in the internal, as well as in its external context. 
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3. Models for the evaluation of vocational training 
programmes (VTP) 

In the following chapter we shall make an initial presentation of a number of different 
types of evaluation as well as defining their significances with regard to the evaluation of 
vocational training programmes (VTP). From among the innumerable possible and 
attempted classifications of the various kinds, forms and types of evaluations, we shall 
be dealing only with those which are relevant for the evaluation of the qualitative aspects 
of vocational training in the context of the CEDEFOP project. 

From a temporal perspective we can start by differentiating between evaluations which 
are orientated towards the future, the present or the past of a given programme. 
Evaluations relating to the future course of a programme and its possible results are 
termed ex-ante evaluations. Evaluations which focus on the course of a vocational 
training programme are called on-going, in-progress or process-orientated evaluations. 
Evaluations which examine an already completed programme with regard to its effects 
are known as ex post evaluations. This division of the evaluations into three groups from 
a temporal perspective is of use inasmuch as it determines the discourse at European 
level. Hence the evaluations of the Structural Funds interventions in the Member States 
follow this pattern. 

From a theoretical point of view the following explanatory presentation takes the CIPP 
model developed by Stufflebeam etal (1971) as a point of reference. In the CIPP-model 
each letter represents a different type of evaluation: C stands for context, I for input, the 
first P for process and the second P for product. Considered methodologically however 
the CIPP model is just one variant among many; the organising principle of the model is 
the use of evaluations to provide contributions towards practical decision-making, which 
is why it is also known as a decision-oriented model. The advantage of this model for 
our purposes is to be found in the fact that it covers the various temporal stages of the 
evaluation. 



3.1 Ex-ante evaluation 

The aim of the ex-ante evaluation is to determine the framework or initial conditions of a 
programme and to assess the objectives which are achievable with the resources 
envisaged; according to the CIPP model this entails the analysis of ‘context’ and ‘input’; 
what are the framework conditions under which the programme is to take place? What 
input is necessary in order to achieve a given goal? 

At the same time the ex-ante evaluation can also formulate the indicators through which 
a programme's effects can be examined after completion (ex post). Freeman and 
Solomon (1984) speak in the context of this function of ex ante evaluations of 
‘evaluability assessment’. In order to do justice to this function, the programme 
objectives must be quantified and/or operationalised into measurable indicators during 
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the course of the ex-ante evaluation. In the same way the extent of the resources 
required for the achievement of the programme objectives must be determined. If the 
level of resources has already been determined, the possibilities for the achievement of 
programme objectives must of course be defined in their dependence on the available 
resources. In the context of the evaluability assessment there should also be some 
evaluation of the possibilities of collating the basic data required for the measurement 
and assessment of the effects of a given programme. 

Also directed towards the future are the so-called ‘feasibility studies’. These are indepth, 
specified ex ante evaluations: context and input are related to each other in a model way 
in a controlled setting and possible results extrapolated. This usually involves playing 
through and comparing different variants or scenarios. 

3.2 Process-orientated evaluation 

With regard to the evaluation of the course of a programme, referred to in the CIPP 
model as the accompanying examination of the ‘process’, it is also possible to 
differentiate between a number of methods and types of evaluation. 

Generally speaking, process-orientated evaluation follows the course of a programme, 
i.e. it mostly takes place in parallel to the enactment of the vocational training 
programme but can of course focus on widely differing aspects, depending on the 
questions formulated. Process-orientated evaluations are interested in the internal 
functionings of a certain programme, in particular the procedures, the modalities, the 
conflicts and comprises which are happening when a programme is implemented. All 
these conditions for implementation are seen as influencing the result. The whole 
process is seen as a means to achieve quality, and a definition of quality can only be the 
result of negotiations between the actors involved. Questions can for example be 
directed towards the documenting of bottlenecks and conflicts in the course of the 
programme, or equally be concentrated on the results achieved or relate to the process 
as a whole; in the latter case we then speak of systems evaluation. 

In as much as the insights gained from the process-orientated evaluation are repeatedly 
communicated to the responsible programme agents in the ongoing programme and 
specific evaluation insights are then taken into consideration subsequently to this 
feedback process in the continuation of the programme, we then speak of formative 
evaluation. This type of accompanying, formative evaluation does not necessarily 
depend on the presence of an evaluator external to the programme enactment, but can 
also be carried out by the programme agents themselves. This is known as action 
research. 

In the context of the evaluation of European programmes, formative evaluations are 
something of a rarity. Programme-accompanying, on-going evaluations have merely a 




12 



18 



summative function here, i.e. identifying the effects of certain conditions and process 
variables on the subsequent achievement of programme objectives. 

3.3 Ex post evaluation 

The term ex post evaluation refers to the analysis of the effects of a completed 
programme, known in the pedagogical tradition of Stufflebeam's CIPP model as the 
‘product’ or also as the ‘output’ in systems theory approaches. The evaluation of the 
results achieved is based on the criteria of the programme's initial or starting conditions 
(context), the originally formulated objectives (target achievement analysis) as well as 
possible (intentional or unintentional) side effects produced by the programme. Ex post 
evaluations are interested in the results of VET programmes making comparisons 
between objectives and achievements. In this case the process which leads to the 
results is considered as a black box. Output-related evaluations examine the adequacy 
and the effectiveness of a certain programme against an a priori definition of different 
quality aspects which can be observed and measured using corresponding indicators. 
In addition it is also possible to carry out cost-efficiency analyses based on ex post 
evaluations which establish the effects of a programme. Such analyses involve not only 
a comparison of the original objectives and the results actually achieved, but also a 
comparison of the various methods of goal achievement. This comparison is designed 
to identify which of the methods used represents the best or most cost-efficient way of 
achieving the envisaged objectives. 

Ex post evaluations are also used to examine the programme as a whole, in order to 
gain information and a value basis for longer-range decisions about whether the 
programme should be continued, altered or even terminated. Does the input need to be 
increased, should the programme objectives be modified or the methods altered? Thus 
from the point of view of their enactment, ex post evaluations are retrospective 
assessments whilst from the point of view of their conclusions they are connected to the 
future. According to their intention, i.e. in as much as their insights are used, they entail 
effects on the future, or rather on the course of future programmes. In this, programme- 
related sense, ex post evaluations also fulfil a formative function on a meta-level, i.e. on 
the level of the programmes’ policy cycle. 

Longer-term ex post evaluations in particular pursue the objective of documenting this 
and other kinds of programme effects beyond their originally formulated objectives, 
including programme effects in other areas of activity and areas of life. In this context we 
use the term impact analyses. 

The following overview contains a summary of the typology of evaluations developed 
above. 
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Table 2: Typology of Evaluations 




pre<eva!uative studies 
in controlled settings 



comparison of methods 
of achieving objectives 



According to the different types and aspects of ex post evaluations the main issues 

relevant for such evaluations can be summarised as following; 

■ The question about adequacy is asking whether a certain policy and its objectives are 
corresponding appropriately (appropriateness) to the social problem or the needs 
which are to be tackled with this policy. 

■ The question about coherence of the programme is asking whether the means and 
the resources of a certain policy or programme are coherent with the objectives 
envisaged? 

■ The question about conformity is asking whether the objectives and the enactment of 
a certain policy is corresponding to certain (pre-determined) quality standards. 

■ The question about effectiveness is asking what the effects of the programme are, 
seen in relation to the objectives envisaged. 

■ The question about efficiency is analysing the relationship between (financial) 
means and resources at the one hand and the effects of the programme on the other 
hand (cost-effectiveness) 

■ The question about the impact of the programme is asking about (indirect) effects of 
the programme relating to the social environment which are not explicitly comprised 
in the objectives. 
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4. Evaluation of vocational training programmes: 
The state of the art in the countries involved 



In the following chapter we give an overview about the culture of evaluation in vocational 
training programmes for each of the countries involved in this study. It will be 
documented the main types and focus of evaluations to be found and the methodology 
which has been used in the reviewed evaluations studies. 



4.1 Belgium 

Apart from the evaluations of the European Structural Funds interventions there is hardly 
a tradition of evaluations in vocational training programmes in Belgium. Some activities, 
developed jointly by the social partners, have been subject of external evaluations. If 
there is any interest into the training programme at all, the main objects of examination 
are the organisation, the implementation process, the promoters, the trainers and the 
trainees. The content of VET programmes has never been the subject of such 
evaluations, it is seen as a ‘black box’. Many evaluations preferably stick to quantitative 
criteria whereas more qualitative criteria referring to the quality of training should be 
taken into consideration. 

In recent years the evaluation of VET programmes has become a remarkable practice, 
but only because of the requirements of the Community programmes. However, quality 
issues are rarely integrated into the evaluation exercises. A meta-evaluation on the ESF 
evaluations in Belgium, done by Delcourt (1993), stressed the weaknesses of the 
available statistical data concerning the main target groups of vocational training 
programmes. The other, few evaluations in the field of VET programmes are not very 
systematic either. If a vocational training programme is evaluated at all, the main focus 
is on products or outcome: How many electricians, joiners or book-keepers have been 
trained? The micro-economic evaluations are mainly interested in variables relating to 
individual characteristics of the participants (age, gender, length of unemployment, 
disability, ethnic minority) determining the placement rates into employment. Technically 
they mainly use descriptive statistics and sometimes probabilistic models for the 
estimation of placement rates after the completion ofVETprogrammes. 

The more the training sectors expands, the less its evaluation is developing, apart from 
the growing self-evaluations of projects. There are internal investigations of bigger 
training organisations going on, but the outcomes are kept confidential and the existing 
reports are not distributed. In principal the results of ex ante as well as of ex post 
evaluations are rarely taken into consideration when designing new programmes or 
politics. 
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4.2 Germany 



In principle all the types of evaluations introduced in the previous chapter are utilised in 
the evaluation of vocational training programme. Especially process-orientated 
evaluations applying quantitative variables like the demographic data of participants are 
traditionally of great importance. Only relatively rare process-orientated studies contain 
data reflecting the personal situation of the trainees. Nevertheless, studies into the 
variables influencing the process of vocational training programmes have gained 
significance in Germany in recent years, particularly in the context of the discussions on 
quality assurance and quality management in training organisations. Furthermore a 
systematic survey of trainees with regard to their subjective evaluation of the learning 
process has meanwhile become almost standard among VTPs. However, there must of 
course be a differentiation between using surveys of this type as feedback for the direct, 
practical course of the training or carrying them out in the context of scientific programme 
evaluations. Altogether, there are hardly any input variables, whether determined by the 
VIP. influenced by the trainees or dependent on the situative conditions of the training, 
which have not been examined in the evaluation of process-orientated evaluations in 
Germany. Furthermore there is a growing tendency to include the trainees subjective 
opinions as a resource for data collation and as an additional indicator for the quality of 
VTPs. 

The placement rates for the participants of all publicly funded programmes for basic and 
further vocational training in Germany is regularly documented by the Federal Institute for 
Labour in the context of its labour-market statistics. This procedure also documents the 
sectors and the branches of the economy in which VTP participants have found 
employment. Those responsible for training and labour-market policies take the 
employment rates as the main criteria for success of VTPs. The fixation on placement 
rates dates from the origins of German labour-market policy (statistics) in the late 
sixties, when structural unemployment was practically unknown, and the VTPs which 
where carried out let to vocational advancement for 90% of the participants. 

Because these general, quantitative data on the effects of VTPs allow relatively few 
practically-oriented conclusions to be drawn regarding the design of VTP. many ex post 
studies have also begun including other, qualitative data on the (employment) situation 
of the participants. This is true for example of the follow-up surveys carried out by the 
large vocational training centres in Germany, as well as of the ex post evaluations 
following on from the European Social Fund interventions. It must be added that more 
recent follow-up studies of VTPs have not been content to confine themselves to 
quantitative methods of data collation on the whereabouts of the participants, but also 
make use of qualitative survey instruments in order to reconstruct VTP participants' 
biographical routes into employment or. as the case may be. into unemployment. 

The meta-analysis of evaluations studies carried out in Germany clearly illustrated that 
there is no lack of relevant research in relation to overall programme cycles. Whilst this 
statement should only be applied with caution to the ex ante phase, it is otherwise true 
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for every stage of the evaluation process and both with regard to the quantitative as well 
as to the qualitative data, whereby the latter serve less as success criteria than to 
provide the basis from which to deduce important indicators for the needs-orientated 
modification ofVTPs. 

It must however be said that the fields of evaluation of vocational training programmes 
and of labour-market policies remain distinctly isolated from each other. The former is 
primarily orientated towards the training process, whereas for labour-market policy 
evaluations it is the integration into employment which enjoys pride of place. Process- 
orientated evaluation of VTPs is largely excluded from labour-market policy ex post 
evaluations and vice versa. In recent years, however, the evaluations of EU Structural 
Funds activities concerning the development of human resources have been an 
important catalyst for bringing together both research strands. 

4.3 Greece 

Until the mid-1980s the volume of vocational training directly oriented to the labour- 
market and its public funding remained essentially marginal and therefore the question 
of any formal evaluation of their effectiveness or quality had never been raised. From that 
time onwards and particularly during the last decade this state of affairs changed 
radically by a considerable expansion of the vocational training activities, fostered by a 
considerable funding through the Greek State authorities and the European Structural 
Funds. ^ This time, however, vocational training was not organised in response to any 
strictly defined labour-market needs but in reaction to the growing rates of actual 
unemployment and to the strong demand for new additional skills, both of which had 
been the result of the structural, technological, and organisational transformations 
which the Greek economy has been undergoing since then. 

In this context the question of quality and effectiveness of training has begun to be widely 
discussed and the evaluation of VTPs have recently gained importance within the private 
and public training sector and particularly among the public authorities concerned. 
Formal evaluation studies into VTPs have therefore a quite recent history in Greece. 
There is only a limited number of evaluation reports available, since many of them are 
thought of as confidential documents for the internal use of the institution concerned 
only. The conceptual approaches and the methodologies which have been adopted in 
evaluations are described quite briefly or not at all. Against this background the current 
state of the art in Greece concerning the evaluation of VTPs is reflected only through a 
small number of available studies which have been reviewed. 

The Greek Manpower Employment Organisation (OAED) can be considered as the 
leading provider of vocational training for unemployed persons or insufficiently qualified 
people. The OAED has not established any formal system for the evaluation of its 

^ The rate and amount of this expansion are being illustrated by the increase of the total expenditure for contiuing 
vocational training in Greece from 0.04% of the GNP in 1987 to 0.24% in 1991 (OECD, Employment Outlook 
1992, Paris 1992). 
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training programmes but it occasionally conducts ex post summative eval^uat.on stt^aies^ 
T^ S s?udv entails an evaluation of the objectives of the programmes embedded in the 
Ltva socTo economic context (context and input evaluation), an evaluation of the key 
^nZlnSion elements (process-orientated evaluation) and an evaluation of the 
:£rs ("odur« and impact evaluation) of the vocational training 

programm6S undar considaration. 

In methodological terms concerning context the OAED evaluation is based on the 
ava^aWe statistical data concerning the main aspects of employment and 
unemployment. The analysis of the training process consists 
analysis of the subjective assessments made by mam 

traininq managers, and key actors on the decision making level. A third cluster of 
evaluation activities is focused on data concerning the employment status of the former 
r" fbeir subjective assessments of the overall "'X 

have participated in to their employability. This data is completed by subjective 
assessments of company representatives employing the former trainees. Furthermore 
the impact evaluation of the training programme consists of an econometric analysis 
using linear multiple regression models, in order to assess the relative importance of 
fectors that are determining the effectiveness of the VTPs in question (O 
evaluations). 



The evaluation studies launched by other major providers of vocational training, too, 
mostly have adopted an ongoing concept, analysing the key input, the process and the 
outcome factors of the training programmes carried out, and it is striking, that 
methodologically they are based quite often on qualitative analysis of subjective 
assessments made by trainees, former trainees, trainers, training managers and other 
relevant actors. Furthermore it is noteworthy that the evaluation of inhouse training of a 
big Greek company uses quasi-experimental approaches to analyse the transfer of the 
skills acquired by the training into the workplace. 

4.4 France 

In France the systematic evaluation of the economical and social policies is a recent 
development. Evaluation is becoming an integral part of public affairs. This is the result 
of an interministerial exercise in the beginning of the 1990s which did not affect 
vocational training policies only but included the whole public sector. In this framework 
evaluation has been defined as a permanent element of pubic policies: ... ‘evaluation of 
a public policy aims to analyse if the administrative, juridical and financial means which 
have been provided for this policy have lead to the achievement of the expected effects’ 
{Conseil scientifique de I'Evaluation, 1991). In France the decentralisation of vocational 
training policies which started in 1993 was a push forward for evaluation exercises, as 
the corresponding law foresaw an evaluation of the decentralised exercises. 
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There is no consistent methodology concerning the evaluation of VET programmes in 
France and the studies which have been reviewed in the framework of CEDEFOP 
research have even less in common concerning the evaluation of quality aspects. 
Concerning the main focus and the methodology used, two different types of evaluation 
can be found in France: 

■ a first type, qualified as product-orientated evaluations, is interested primarily in the 
results of VET programmes making comparisons between objectives and 
achievements. In this case the process which leads to the results is considered as a 
black box. These output-related evaluations examine the adequacy and the 
effectiveness of a certain programme against an a priori definition of different quality 
aspects which can be observed and measured using corresponding indicators. 

■ a second type of evaluation, which can be qualified as process-orientated, is 
interested in the internal functionings of a certain programme, in particular the 
procedures, the modalities, the conflicts and comprises which are happening when a 
programme is implemented. All these conditions for implementation are seen as 
influencing the result. The whole process is seen as a means to achieve quality, and 
a definition of quality can only be the result of negotiations between the actors 
involved. 

In France the development of process-oriented evaluations is forced by the distribution 
of guides of good practice often used as ‘cBhiers de chdrge', which have been 
elaborated at national level with the aid of training experts. Some evaluations sought to 
prove if these cahiers de charge had been applied by the training organisations. Most of 
the French evaluation reports do not deal with the analysis of cost-effectiveness issues. 

More recent developments are directed to goal-free evaluations (according to 
Mr Scriven), focusing on questions like the internal and external coherence of the 
programme as well as its intended and non-intended results. Furthermore there is a 
growing tendency for self-evaluations to be in parallel with the enactment of training 
courses carried out by many of the training promoters themselves. In training for 
disadvantaged groups self-evaluation is also seen as a means of empowerment of the 
trainees. 

The scientific council has published guidelines for evaluations including conceptional 
and methodological proposals. According to this guidelines, two approaches to analyse 
quality aspects of vocational training programmes can be distinguished. The first 
approach consists in the elaboration of indicators to measure the quality of a certain 
programme, the second approach goes along with the piloting of quality aspects in 
training activities. Here, the main focus is on the question how quality is assured, and 
what are the means to assure quality. 

Whereas in Belgium longitudinal studies have not been carried out yet in the field of 
vocational training and integration into employment, in France (Aucouturier, 1994) and 
Germany (Schomann 1994) there are already some experiences with this approach. 



4.5 Portugal 



The tradition of the evaluation of VTP programmes in Portugal is mainly outcome- 
orientated, it is connected to the evaluation of results of students and trainees. Since the 
1960s, there has been a tradition of evaluating education and training at a macro-level: 
performance and efficacy/efficiency of educational systems are the main worries of 
political thinking. Evaluation of training programmes started later and it took longer to 
involve the research community, which is usually more interested in education than in 
training. At the micro-level, the object of evaluation has been the measurement of 
learning/knowledge. With the growing complexity of vocational training, programmes are 
no longer seen in their univocal relationship trainer/trainee but tend to create a logic of 
their own. Processes and results are seen as a part of the whole. The analysis of 
vocational training tend to be systemic, contextualised, in such a way the processes that 
occur inside the training systems are understood as an answer of the actors towards 
the different challenges they receive from the decision centres or from the local and 
regional socio-economical environment. 

There is not yet a generalised culture of evaluation in Portugal. The integration of 
Portugal in the EL), a closer and systematic contact with different realities and points of 
view, and the need to answer to new demands started to make a difference in the 
evaluation of vocational training programmes. These new needs are related to the 
allocation of European funds and their application/use in the areas of education and 
training. The policies of employment and training together with education and scientific 
research allocated 90% of the ESP in Portugal. Since it is a contractual obligation to 
evaluate European programmes, this has led to an investment in the search of models 
and methodologies of evaluation. 

In spite of that, the relationship between objectives / results (effectiveness) and costs/ 
benefits (efficiency) is still the organising concept in many of the available studies in 
Portugal. However, this approach makes it difficult to get information about the studies 
and the results obtained. The published studies are scarce and the access to written 
reports is really limited, regardless of whether the entities are private or public. 
Furthermore evaluation is limited by the nature of information available. Many studies 
are damaged by the absence of organised and systematised information. ‘As a result of 
the difficulties faced, we consider to be urgent the conception and seting up of systems 
of information on the evaluation ...’ to be made in the future. This statement was written 
in 1994 (Department of Statistics of the Ministry for Labour and Social Security), but 
recent studies still refer to the same problem. 
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5. Evaluations of quality aspects ot vocational training 
programmes 

This section is concerned with the evaluation of quality aspects in vocational training. In 
this context it is to examine whether the essential criteria for quality as defined by the 
present CEDEFOP study are met by the evaluation reports which have been reviewed in 
the countries involved and whose methodological standards have been applied to 
investigate the research questions concerned. By presenting the findings this chapter is 
following the different types of evaluations as described in chapter 3. 



5.1 Evaluation of context 

Context evaluations of VTPs should contain an examination of the target group and its 
starting conditions, of the resources available and the funding conditions which are 
available as well as of the objectives connected with the programme. In addition, as a 
result of context evaluations, the training programme objectives should to be 
operationalised and the (implicitly) evaluation criteria should be formulated. 

In reality, context analyses of VTPs are mainly focused on a critical examination of the 
labour-market structure at the national levels and its recent changes. Mostly they are 
based on the available employment and unemployment statistical data that have been 
recorded by the national labour force surveys and by relevant studies. Some Greek 
studies contained reviews of the training policies and the programmes carried out in the 
European Member States by annotating comparative comments on their advantages 
and disadvantages as well as on their transferability in the given national context. 
Against this background an overall appraisal of the objectives and the intended 
outcomes of the envisaged training programme against the particular socio-economic 
context could be stated. 

For Portugal it was considered that the needs of the trainees were rarely taken into 
account, because it is taken for granted that the needs of the trainees cannot be different 
from the needs of the labour-market. Thus the evaluation studies refer to the needs for 
training, which are defined at the macro-level, through sectoral or regional surveys and 
forecastings and/or according to the objectives of the European Funds. Whereas these 
findings are relevant for training programmes for unemployed people, a totally different 
approach was reported for the planning of continuous training activities inside a big and 
quality-certified enterprise. Here, the trainees needs are defined through interviews and 
questionnaires addressed to managers, workers and clients. Additionally there were 
internal auditors, to observe and evaluate the competences and the difficulties of the 
workers at their workplace. The results of such context evaluations are related to the 
strategic aims of the enterprise and integrated into the planning of training activities. In 
the decision on implementation of training activities the training department of the 
enterprise is a mediator between the managers, the requirements of the enterprise and 
the needs of the workers. 
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In general it must De seen as a weakness of context evaluations of VTPs that the 
characteristics of specific target groups are not analysed more precisely. Moreover, 
most of the context evaluations are not detailed enough to come to conclusions 
concerning the organisation of vocational training for the target groups in question. The 
overall design of vocational training programmes therefore normally does not reflect the 
social conditions of the target groups envisaged. A French synthesis report (Bouder et 
al., 1994) reviewed studies about young people, but none of the studies contained an 
explicit description of the special needs of young people with difficulties concerning 
vocational integration. Solid criteria for the identification of ‘at risk’ target groups are often 
lacking and examinations into the interface between training needs of young and risky 
people and the existing vocational training system have been neglected so far. 

An ex ante evaluation carried out in the framework of the European Social Fund objective 

3 (DULBEA - Joyeux, Meulders, Ruz Torres 1993) has shown that women, elderly and 
isolated people, people who care for other people, and those with low qualifications are 
particularly exposed to poverty. Given the difficulties these groups have in participating in 
VET programmes, another study (FTU - Huon, Pjetri 1993) proposed a broad approach 
to tackle social exclusion and poverty, which includes activities to promote vocational 
integration, councelling and motivation, as well as cultural and social integration 
(housing) activities. 

Other context-orientated ex ante studies conducted in the framework of the ESF objective 

4 used econometric models to anticipate the development of employment in 
25 economical sectors (Fierens, Meulders, Sekkeat 1993). For some sectors the 
requirements concerning vocational training have been identified. In general, however, 
these projections were mainly theoretical and failed to be detailed enough to include 
conclusions on the perspectives and the design of VET programmes. The main 
weakness of such prospective appraisals is that sectoral employment forecasts are 
usually translated into numbers of posts but not into skills and competences. 



5.2 Evaluation of input 

Input analyses of VTPs are mostly restricted to overall appraisals of the objectives, 
intended outcomes and funding conditions of the training programmes in question, 
done against their particular contexts tending essentially to be a kind of training policy 
evaluations. Some studies with input analyses include an inquiry of the formal decision- 
making and planning processes determined by the organisational or the operational 
structures of the training providers considering them as crucial factors affecting the 
quality of its training offer under evaluation. 

Most of the input analysis which was reviewed focused on the individual training 
programmes in question and did not employ any comparative approach. The factors 
inquired into included: 




22 



28 



■ adequacy of the training objectives and contents in relation to the requirement on the 
labour-market; 

■ appraisal of the effectiveness of the training plans, contents and methods in 
achieving the intended outcomes; 

■ experience and organisational status oftheVTP promoter; 

■ suitability of the training materials and tools; 

■ quality of the premises, equipment and training conditions; 

■ effectiveness of management and supervision structures of the training. 

It became obvious, that clearly defined quantitative input data are an important yardstick 
for the evaluation of the training process and its outcomes. Evaluations of public 
employment programmes done by the French audit office have shown, for example, that 
the majority of the beneficiaries in such programmes did not belong to the target groups 
which had been envisaged and prioritised by the programme. Similar findings are 
reported from Portugal. Inconsequence, proposals have been made to introduce some 
quota figures for certain target groups and thus supporting their proportional 
participation in VET. 

Another input analysis has shown that the unemployed with children, if they are on their 
own, often experience difficulties with child care facilities when wishing to start up with 
training or employment. In this rare case, the design of the training courses took into 
consideration the special needs of the those trainees by establishing accompanying 
child care facilities. 

However, the lack of commonly accepted and applied quality criteria as well as the 
variations in the evaluation methods employed by the relevant studies objectively limit 
their conclusions concerning the relevance of these input variables to the individual 
training programmes considered and do not allow any further contribution to the quality 
of training programmes at a general level. 

In addition, since the analysis of input conditions of the training programmes are at 
most part of ex post evaluations their conclusions cannot be formatively utilised to the on 
going process of the training in question; they are generally of use for the reform of the 
next training programmes implemented which might, however, operate under quite 
different conditions. 

5.3 Process-orientated evaluations 

Process-orientated evaluations are not primarily interested in the results of the training 
activities but in the training system and the proceedings which are happening along the 
enactment of the training courses. Amain question concerns the appropriateness of the 
process according to what has been formulated through context and/or input evaluations 
as demands or objectives. 
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During recent years it has become clear that vocational training is only one element in a 
development which should include motivation, orientation, assessment, training and 
assistance with job research and placement. Given the fact that the process of 
vocational integration thus consists of several steps it seems all the more necessary to 
evaluate the interaction between these different elements leading to vocational 
integration. Such process-orientated evaluations (will) not only lead to a better 
understanding of the relationship between the process and the results obtained, the 
existence of certain organisational modalities in the process of vocational integration 
must also be seen as criteria of quality. 

The evaluation of the vocational training process itself relates to all the factors which can 
influence the course and the success of a training programme during the process of its 
implementation. In the following we differentiate between the conditions brought into the 
process by the trainees, the trainers and different elements of the vocational training 
process itself. 



5.3.1 The conditions of the trainees 

With regard to the conditions brought into the process by the trainees, the 
documentation of the following variables belongs largely to the standard repertoire of 
process-orientated evaluations: 

■ demographic data such as age, gender, marital status, 

■ data on the pre-programme educational achievements, i.e. initial school and 
vocational qualifications as well as - if applicable for the relevant target group - the 
length of vocational experience, 

■ the length of unemployment in programmes for the benefit of unemployed people, 

■ other relevant biographic data depending on the group the programme is tailored 
towards (e.g. single parents, the disabled, migrants, drug addicts, etc.), 

■ records of the refused applicants to training programmes, which normally are not 
available to evaluations but are worth considering. 

Only rarely do studies on the input conditions at participant level also contain data 
relating to the participants' own motivation and interests, although there is no doubt in 
the relevant theoretical literature that participants motivation is a decisive condition for 
learning achievement. Analyses of access routes sometimes provide information on the 
trainees' motives for participation in a particular programme: hence it will probably make 
a big difference to the result if participation in a VTP can be traced back to the 
participant's own initiative or whether it comes about through lack of alternative or even 
in response to the demands from social allowance authorities. The relatively little 
attention paid to the subjective initial situation of VTP participants in evaluations could 
generally speaking be taken as an indication of the fact that participant's needs are of 
relatively little significance in the original design of training programmes. 
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5.3.2 The trainers qualifications 



Of course the training personnel and its qualifications must be regarded as another 
important part of the training process and hence as one of the factors amenable to 
influence. If the evaluation of trainers and trainers behaviour nevertheless appears as 
an independent category this is due to the fact that this ‘decision variable’ has been 
accorded a great deal of significance in some relevant studies on the quality of the 
training process. The variables examined include: 

■ the trainer's formal qualifications and professional expertise, 

■ teaching skills and communication abilities (for example the trainer's capacity for 
empathy), 

■ the quantitative participant-trainer relation in the VTP, 

■ the trainer's experience with the target group as well as with the training methods 
used. 

It has to be underlined that some of these variables can be examined according to 
objective data (formal qualifications) whereas other are examined on the basis of 
subjective assessments. Their validity is therefore heavily dependent on the 
methodology and the instrument applied for their collection and recording. 

Most of the factors dealt with up till now which are brought into the training process as 
condition-related ‘constant variables’ from the participant or trainer side can either not 
be influenced at all (age, sex) or influenced only to a limited extent (qualifications of 
trainers) and must hence be considered as constants in VTP designs. However this 
does not apply to the process conditions ofVTPs, for example the content and methods 
of the training courses, which can be seen therefore as ‘decision variables’. These can 
be influenced by those responsible for the training, and are thus subject to variation and 
alteration. 



5.3.3 Evaluation of situative process variables 

Studies on the process on such decision variables of VTPs have gained in significance 
in recent years, particularly in the context of the discussions on quality assurance and 
quality management. The following factors have come under examination with regard to 
their effects on VTP results: 

■ relevance of the training contents in relation to the needs and the expectations of the 
trainees, 

■ conformity of the training methods to the training contents, 

■ adequacy and effectiveness of the time schedule, 

■ training location (comparisons between programmes based in firms and centre- 
based training programmes), 

■ possibilities for participants to influence the course of the training through 
participation in decision-making processes (empowerment approaches). 



■ comparisons between various training methods (individual and group tuition, team 
learning, etc.), 

■ examination of the learning atmosphere (supportive or cooperative climate in 
comparison with competitively orientated settings). 

Especially the introduction of the ISO norms into the training sector resulted in a growing 
evaluation of process criteria. The implementation of the ISO standards assures that 
certain quality criteria are respected. The evaluation of quality aspects of training 
sometimes include an analysis of the subjective assessments made by the former 
trainees about three months after completion of the training course. This analysis is 
focused on to the actually acquired competences and the extent of their applicability and 
transfer at the workplace of the former trainees. 

In Greece, the process quality of continuous vocational training is evaluated applying 
methods and criteria derived from the philosophy and the objectives of the participatory 
training approach that assigns to the individual trainees a leading role in the training 
process (Rogers, 1993). The criteria used for the assessment of the training quality are 
determined by the trainees themselves according to their subjectively perceived needs 
and perspectives. Here, the evaluation task consists of a primarily qualitative analysis of 
the subjective views expressed and the assessments made by the trainees. 

Other qualitative examinations included in process-orientated evaluations focus on 
examinations of individual pathways into employment. Such approaches analysing the 
needs and integration pathways of unemployed persons use biographical analyses in 
combination with critical reviews of the existing training systems. Other quality aspects 
analysed in the frame of process-orientated evaluations comprise the kind of acquisition 
of skills, the quality of orientation and information of trainees, their level of qualification 
when starting with the vocational training and their level of skills when leaving the 
training system. Evaluations with a focus on pedagogical issues have analysed the 
relationships between contents and objectives of training courses, the principle of 
individualisation, or the importance of social assistance to trainees. 

However, the lack of commonly accepted and applied quality criteria as well as the 
variations in the evaluation methods employed by the relevant studies limit objectively 
their conclusions to the individual training programmes considered and do notallow any 
further contribution to the quality of training programmes at a general level. 

The more systemical approaches to process evaluation of VTPs are dealing with the 
relationships and the networking activities between the socio-economic actors 
(enterprises etc.) and the local training organisations. Such evaluations focus, for 
example, into the question if a job research assistance and a placement service is 
provided by the training organisation or if regular agreements are made between 
enterprises and the training body. They could analyse the changing roles of training 
centres, local authorities, and small and medium enterprises (SMEs). The challenge of 
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such evaluations consists in their possible contribution to overcome the institutional 
limitations and restrictions of the current training system. 

However, when analysing all these different elements which in principle could influence 
the quality of training activities, it should not be forgotten that participation in highly 
adapted, individualised training circuits for certain individuals could also mean 
segregation from integration into the employment system. 

5.4 Product-orientated evaluation 

In the product-orientated evaluation of vocational training programmes it is the learning 
achievements of the participants which are of first and central importance, serving as 
they do as a criterion for goal achievement and hence also as a criterion for the 
examination of the effectiveness of the programme. Further criteria for product-orientated 
evaluations are the adequacy of the training, the effects of learning transfer into the 
working environment, the access rate into employment, and the quality of the job 
obtained. 



5.4.1 Assessment of learning achievement 

Traditionally the assessment of learning achievement takes place at the end of a 
vocational training measure through examination of the increase in knowledge of the 
participants. This is, for example, especially the case in apprenticeship programmes for 
young people. In this context largely objective forms of performance measurement are 
traditionally used. Usually the net training outcomes should be evaluated against the 
objectives set out at the beginning of the VTP. If this evaluation takes place at the end of 
the training, its results of course can no longer contribute formatively to the ongoing 
process but are at most of use to the following process generation. In order to influence 
the ongoing learning process via feedback, examinations of the participants' 
performance abilities have to be carried out at intervals or on a continual basis. 

However, the measurements of learning achievement allow few conclusions to be 
drawn about the qualities in the training process which were particularly beneficial for 
the participants. And it is precisely the modern key qualifications such as aptitude for 
team work, creativity, or problem solving ability which can scarcely be measured by 
objective assessments. 

For these reasons the qualitative analysis of the subjective significance of certain 
variables in the training process is playing an increasing role in process- and product- 
orientated research. In some of the countries involved in the present study a subjective 
assessment of certain aspects of the training process by the trainees has now become 
almost standard among VTP evaluations. The adequacy of the content and the suitability 
of the training process, for example, can be evaluated on the basis of the conclusions 
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yielded by a descriptive analysis of the subjective assessments made by the trainees. In 
principle surveys of participants can refer to all the variables brought into the training 
process by the programme's target group orientation, the situative conditions or the 
training personnel. However, most participant surveys are not differentiated enough to 
be able to encompass these variables in their entirety. Questions are often asked in a 
relatively general way regarding the participants' 'satisfaction' with certain conditions, for 
example with the quality of the teaching materials, the learning atmosphere or the 
degree to which it was possible for them to influence the design of the training process. 
In addition participants may be asked to give an assessment of their trainers. 

A mixture of objective and subjective assessments is necessary, too, for analysing the 
drop-out rate, i.e. the number of participants who broke off early from their VTP or ended 
it unsuccessfully. To draw conclusions about the conditions which may have led to 
dropping out, a record has to made not only of the number of drop-outs but also of the 
reasons for premature termination. The results of these subjective assessments may 
allow conclusions to be drawn about specific conditions in the training process which 
are incompatible with the needs of the participants. 

Although the participants' subjective assessments cannot provide the sole guideline for 
the design of future training processes, given that they sometimes merely reflect 
personal preferences and that objectively established aims, such as the fulfilment of the 
requirements of a specific vocational profile must also be achieved within the framework 
of VTP evaluations, these qualitative surveys do nevertheless provide important 
indicators for the adaptation of the VTPs to the needs and the requirements of the 
participants. The question which is important for any programme evaluation, namely 
whether and to what degree a programme has achieved its objectives, will certainly have 
to be answered primarily on the basis of objective data. And yet the subjective 
assessments of programme participants can make a comparatively greater contribution 
to the formative development of programmes and the modification and fine-tuning of 
subsequent generations of programmes in accordance with the requirements of their 
target groups. 



5.4.2 Evaluation of the learning transfer 

Another criterion for the measurement of the outcome or product of training processes 
has been introduced into the debate by examinations of learning transfer. In the context 
of vocational training, learning transfer is understood as the application of what has 
been learnt to new situations at the working place. Learning transfer research not only 
asks about the learning achievements attained through participation in a VTP; rather 
interest is concentrated on the usefulness for everyday vocational practice in the 
enterprise which emerges as a consequence of VTP participation. Research of this kind 
is most often undertaken in the field of in-house further education, which the larger firms 
usually organise themselves. Hence firms are not particularly interested in whether their 
employees attain objective learning achievements and which kind of (generally 
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recognised) certificates they acquire through participation in continuous training. Of far 
greater importance for the enterprises is the usefulness of what is learned and the 
degree to which it can be practically applied in everyday work. 

As reported from Germany, Greece and Portugal, studies evaluating in-house training in 
big enterprises were using this outcome criterion by means of subjective evaluation 
methods (asking participants in further training whether they are able to use the 
knowledge they gained in their normal area of work) and by objective evaluation 
methods (assessment of the participant's way of working some time after completion of 
the training). 

The main challenge for this type of evaluation studies seems to be the establishment of 
counter-factual conditions against which the net impact of training on the learning 
transfer is appraised. This element considered either by subjective or by objective 
evaluation methods remains problematic. In the cases where subjective evaluation 
methods are employed the counter-factual conditions are at most implied by the ex- 
trainees' subjective assessments concerning their transferred learning attributed to 
training that is however based on personal and not clearly specified criteria. On the other 
hand, in the cases where objective evaluation methods are employed the counter- 
factual situations are usually established by comparisons between before vs. after 
training and/or between training participants vs. non-participants demonstrated 
individual competencies. The credibility of the conclusions yielded by both comparisons 
is of course questionable since, not meeting the necessary empirical control 
requirements, they do not account for the influences of any other factors than training on 
the learning transfer demonstrated by the former VTP trainees. 



5.4.3 Evaluation of employment prospects 

In the following section it remains to be examined the extent to which further criteria for 
an outcome-orientated evaluation of VTP can be included in addition to learning 
achievement and learning transfer. This will mean examining whether and to what extent 
the question of the integration of the VTP participants into employment represents an 
additional criteria which can be used to measure the effects of VTPs. 



Vocational training programmes do not only aim to generate a specific learning 
achievement but are also concerned with integrating their participants into the labour- 
market. In other words, the ideal objective of most VTPs is the achievement of full 
employment for its participants on the general labour-market as a result of their 
participation in the training programme. The achievement of this objective is examined 
in the context of product-orientated ex post evaluations. Generally speaking the 
‘outcome’ criterion ‘integration into employment after completion of the VTP’ can be 
described by the following indicators: 

■ employment quota (proportion of persons entering normal, socially insured 
employment at least once during a given period). 
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■ average duration of employment for these persons, 

■ employment quota at a given point in time (e.g. two or four years after completion of 
the VTP). 

This methodological procedure to collect those data may also document the areas of 
the economy in which VTP participants have found work. In addition these data may be 
evaluated in a regionally differentiated way (according to certain regions). This enables 
conclusions to be drawn on employment prospects in specific vocational sectors and 
regions, and the data may be used also to direct the supply of VTP in specific vocational 
areas. If, for example, participants in VTPs in commercial professions remain over 
proportionately unemployed in a certain region, the supply of VTP in this area can be cut 
back through the regional public authorities. 

It should be evident that as long as unemployment exists to a large and structural extent, 
not all VTP participants will be able to enter employment. Nevertheless, the employment 
integration quota is still taken by many of those responsible for labour-market policy as 
the main criteria for the evaluation of the success of VTPs. 

On the other hand the explanatory value of the employment integration quota as an 
outcome criterion for VTPs has had to be increasingly restricted in the face of research 
which clearly and consistently showed the existence of a connection between 
employment integration quotas and rates of unemployment as an indicator of the 
situation on the labour-market at the time. The employment quotas after VTPs 
documented in numerous studies correlated closely with the state of the economy and 
the situation on the labour-market. 

In addition it was also possible to detect differences between the various target groups, 
and to this extent it is also possible to determine factors for success in the context of the 
evaluation of labour-market policy measures. One of the main findings has been that 
identical training measures have led to different results concerning integration into 
employment for male and female trainees, for young trainees without working 
experiences or for experienced participants, for people with high and low qualifications. 
With higher skills the opportunities forgetting a job are better, but there is no guarantee 
of finding a stable job quickly (Vanheerswynghels, 1996a). When participating in 
comparable VETs women not only have lower placement rates, but also more often gain 
part-time jobs or time-limited contracts. Thus there is evidence which shows that the 
greater the number of so-called risk factors or characteristics militating against 
placement (female, advanced age, low or inadequate qualifications, disability, etc.) in a 
given group, the lower the rate of employment integration. 

The placement studies which have been reviewed produced some important 
methodological insights, too. It has been shown that the access rate to employment 
also depends on the timing of the participant survey following the end of the VTP. Given 
that the transition to employment seldom takes place directly after a VTP and that a 
certain time for looking for employment ought therefore to be taken into the calculation. 
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the surveys which are carried out a certain time after the end of the VTP produce a better 
rate of integration. To determine the long-term effects of vocational training additional 
follow-up studies can be conducted at several intervals. 



5.4.4 Quality aspects of employment 

Because the general, quantitative data on the employment rates allow relatively few 
practically-oriented conclusions to be drawn regarding the design, of VTP, many 
outcome-orientated ex post studies have also begun including other, qualitative data on 
the (employment) situation of the participants. This is true for example for the ex post 
evaluations following on from the European Social Fund interventions. When inquiring 
into the quality of jobs obtained after completion of vocational training different criteria 
can be taken into consideration. 

Important efforts can be seen to link the level of qualifications achieved in the VTP with 
the type of employment currently occupied. Compared to the research tradition in Anglo- 
Saxon countries, changes in remuneration are analysed very rarely. The level of 
qualification is not related to the level of remuneration but to the socio-professional 
status of the employee. These include publicly subsidised and non-subsidised 
employment, time-limited and not determined employment and job-seeking. In the 
surveys there have been distinctions made according to the following categories: 

■ employment situation: 

paid employment, self-employed, permanent/temporary contract, in publicly 
subsidised employment, 

■ non-employment situation: 

unemployed, participation in a new VTP, domestic work, military service or its 
equivalent in the civilian economy, retirement. 

With regard to the quality of employment, further questions are asked about whether the 
employment is full or part-time, and whether the extent of the employment corresponds 
with the current requirements of the former trainee. In addition ex post surveys also ask 
questions to those in employment about the following characteristics of their workplace: 

■ employing firm active in which sector of the economy, 

■ size of the firm in which employed, 

■ current activity and its relation to the VTP attended (continuity or discontinuity), 

■ usefulness of the qualifications acquired for current work (standard of employment in 
relation to training, possibilities for learning transfer), 

■ current income, 

■ vocational prospects (as an indicator of satisfaction with current work), 

■ duration of unemployment between completion of VTP and commencing 
employment, 

■ way in which employment was found. 



American and Swedish studies have more often quoted the increase in income 
achieved following attendance in a VTPas the only (objectively measurable) criterion for 
the evaluation of the qualitative effects ofVTPs. 

As a rule the follow-up data on the current employment situation are taken in relation to 
the data concerning the participants, so that it is possible to make differentiated 
statements about the which target groups have been particularly successful and which 
target groups still require further action on their behalf. It must be added here that in 
particular the more recent follow-up studies have not been content to limit themselves to 
quantitative methods of data collation on the whereabouts of the participants, but also 
make use of qualitative survey instruments in order to reconstruct VTP participants' 
biographical routes into employment or, as the case may be, into unemployment (cf. 
Plath etal 1995). From these data it is possible to draw conclusions about the fields of 
vocational training which could be selected more or less often in the future, in order to 
improve employment prospects for the VTP participants. In addition the information 
acquired in this way also illustrates which institutional forms of VTP are more, and which 
are less successful. 

In part the ex post surveys of previous VTP participants also examined social integration 
criteria external to the work process. For example, a Portugi^ese evaluation study stated 
as a conclusion that the change in the structure of qualifica'tions caused by training was 
almost meaningless, whereas the effects were very important on the social level. 
Evaluations including this kind of outcome data are analysing the following factors: 

■ type of leisure activities undertaken by the former trainee, 

■ quality of social relationships at and outside of the working place, 

■ type of accommodation before and after participating in the training, 

■ general level of satisfaction with life (before and after participation in training). 

Such criteria are generally cited in order to draw comparisons between participants who 
have found employment following on from a VTP and those who have remained 
unemployed. 



5.4.5 Longitudinal studies 

Product-orientated evaluations could be ameliorated through longitudinal studies 
following the itineraries of the participants. Such studies could reveal the kind and length 
of phases which build the process of transition into employment. They aim to identify the 
individual pathways of trainees in a given period. Compared to traditional follow-up 
studies longitudinal studies use a more descriptive approach which is put into practice 
either by accompanying a certain panel over a period of time or by retrospective 
inquiries. Individuals can be grouped according to variables characterising their socio- 
demographic conditions, their societal situation or the degree of their vocational skills. 
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Longitudinal studies may be combined with a control group or comparison group 
approach. They may elucidate the role of different mechanisms or instruments which 
make their contributions to the pathways of the trainees into employment and thus may 
identify barriers or accelerators for biographical development. 



6. Summary and conclusions: The dynamic relationship 
between process and product 

Within the framework of this study, a certain number of evaluation studies was reviewed 
in five European countries (Belgium, Germany, France, Greece and Portugal). The 
purpose of this meta-analysis of evaluation studies was to analyse the methods and 
practices in the evaluation of quality aspects in vocational training programmes (VTPs). 
The reviewed studies had to do with VTPs for the unemployed and, on a lesser scale, 
programmes for the continuing training of employees. Most of these VTPs were at least 
co-financed from public funds. 

Although relatively large number of evaluation studies has been conducted everywhere, 
not least thanks to the impetus from the programmes co-financed from European funds 
but several studies - a number of studies particularly in Portugal and Greece - did not 
necessarily lend themselves to closer analysis. In some cases the methods used were 
regarded as private know-how and were not, therefore, mentioned in the published 
studies. In others, results particularly those of a critical nature were not always 
published for reasons of opportuneness. A culture of transparent communication and 
open debate of the methods and results in the evaluation of VTPs is to be found only to a 
limited extent in most countries. The results of the evaluation of the further development 
of programme practice are also subject to constraints when it comes to their unbiased 
examination and utilisation. The critical discourse between researchers and 
programme evaluators on the one hand and the people responsible for the 
programmes in political and administrative circles on the other is in need of further 
development. Programme evaluation must not be reduced to an affirmative function for 
political and administrative circles. This task can only be properly undertaken if there is 
sufficient critical distance, otherwise the application-related potential of evaluation to 
improve the quality of VTPs will not be fully tapped. 

In terms of methods the reviewed evaluation studies varied considerably. They are still 
very far from using comparable methods. This has firstly to do with the subject of the 
studies and the many different variables which influence the course and the results of 
VTPs. Secondly, the methodological heterogeneity also has to do with the fact that a 
generally accepted, uniform inventory of methods for the evaluation of VTPs has only 
gradually emerged in the last few years. 

If we examine this development more closely, we can initially distinguish between two 
different and indeed contradictory types of evaluation. The first has its roots in university 
tradition and is frequently conducted in this institutional environment. It is called applied 
social sciences. It is linked to major ambitions in respect of adhering to scientific 
standards, which explains why evaluation of this kind is often criticised for being too 
removed from practice. The second type of evaluation stems from management and the 
implementations of programmes. In this case the training function is to the fore. 
Evaluation is viewed as an integral part of the management of training programmes and 
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is therefore less interested in generally valid results than in direct improvements to the 
programme. 

In order to systematically classify the many diverse methodological approaches 
identified during the meta-analysis of VTP evaluations and, at the same time, to 
structure the evaluation field of this study, use was made of the CIPP model of 
Stufflebeam et al (1988). Based on this, a distinction was made between context 
evaluations, process-orientated and product-orientated evaluations. 

Product-orientated evaluations are designed to measure the gap between certain 
objectives of the programme and the results or products generated in the course of the 
programme. In projected/actual comparisons, a distinction is made between the target 
objectives and the products generated. A high degree of goal achievement in this 
evaluation approach is equated with a high standard in the reviewed VTPs. In an ideal 
situation the objectives are operationalised prior to commencement of the programme 
in terms of quantity, which means that the degree of goal achievement can be clearly 
measured later on. Such operationalised objectives may be the number of trainees 
from specific target groups (e.g. proportion of men and women) who can be reached by 
a VTP. Objectives of this kind can describe the skills and formal qualifications to be 
obtained by the trainees in the course of the programme and they can anticipate the 
employment rates which the trainees are to achieve on completion of that programme. 
Accordingly, a distinction will be made in respect of the products of VTPs between output 
(number and type of people involved), results (number and type of qualifications 
obtained) and outcome (direct employment effects of the programme). 

Process-orientated evaluations are concerned with the implementation of the 
programme itself, with the procedures, modes, agreements, conflicts, negotiations, 
arrangements, translations which are involved during the practical course of VTPs and 
which crop up between the players involved. Process-orientated evaluations, therefore, 
have to do with the organisation and the procedures of VTPs. By way of tradition, 
process-orientated evaluations are rather formatively orientated and it is, therefore, very 
difficult to compare their methods and results. However, with the increasing introduction 
of quality standards, e.g. ISO standards, the methods of process evaluation have been 
placed on a more uniform foundation since compliance with specific organisational and 
procedural standards is increasingly viewed as a guarantee for the quality of training 
programmes. 

Context evaluations have to do with the framework conditions under which VTPs are run. 
This means economic, legal and social apsects on the social macro level as well as 
institutional and organisational aspects on the meso and micro levels of the promoter 
organisation. More or less extensive analyses of the context of VTPs are almost always 
an integral part of each evaluation irrespective of whether this is a process- or 
production-orientated approach. However, there are also context evaluations which are 
independent of the product and process. Some examples are prospective analyses of 
the regional development of labour demand or the need for specific occupational 



qualifications in specific branches and sectors (agriculture, shipbuilding, textiles, etc.). 
Analyses to determine the qualification requirements and training needs of specific 
target groups (e.g. disadvantaged young people, migrant workers) also come into this 
category of evaluation studies. 

The meta-analysis of VTP evaluation studies conducted in five European countries 
clearly illustrated that there is a lack of relevant research in respect of the overall vocatio- 
nal training programme cycle. There are scarcely any studies which look at the links 
between the context, process and product ofVTPs. There are many studies focusing on 
single elements from this cycle. There are, for example, studies on training needs. 
There are other studies which are concerned with the organisation of vocational training, 
and there are studies on the integration of trainees into employment, but only few of 
them cover the whole VTP cycle with its various elements. 

The context evaluations examined within the framework of this study illustrate that only 
rarely have there been successful attempts to look at the overall situation involving the 
‘objective’ training needs and the subjective training requirements of the potential 
trainees. In the results of context evaluation on the macro level, training needs are 
frequently defined solely in terms of the (future) demand on the labour-market. Context 
evaluations, which focus on the initial social conditions and the subjective training of 
specific target groups, therefore, more frequently neglect the objective trends on the 
labour-market. The result is that VTPs are primarily analysed in terms of their suitability 
for specific target groups. One exception are evaluations of in-company continuing 
training programmes of large companies which systematically endeavour, both in the 
design of training but also in the evaluation of the results, to consider the interrelations 
between the objective requirements of the in-company work processes and the 
subjective interests and experiences of the employees. 

In respect of process-orientated approaches there are hardly any variables, whether 
influenced by the composition of the participants, dependent on the process of the 
training courses, or the qualifications of the training personnel, which have not been 
examined in the VTP evaluations. However, many of these important process variables 
have not been analysed in a systematic way. This, of course, reduces the impact and 
the general validity of the results of such studies. In process-orientated evaluations 
there is a certain tendency to include the participants' own subjective perspectives in the 
evaluation of the process and this indeed can provide important pointers towards the 
qualitative modifications of VTPs. This tendency should not, however, hide the fact that 
analyses on the subjective needs and motivations of the participants still receive too 
little attention in the planning of vocational training programmes. 

One overall finding of this study is that the fields of vocational training research and 
labour market policy research are still distinctly isolated from each other. This has to do 
with the fact that both prefer different evaluation approaches. The former is primarily 
orientated towards the evaluation of the training process and its internal quality criteria, 
whereas products like the employment rate of the trainees are ignored, if not even 

o 

ERIC 



36 



42 



intentionally rejected as quality indicators. Training research is interested in the process 
but not at all in the product. By contrast, for labour-market policy research it is integration 
into employment which enjoys pride of place. In terms of labour-market policy discourse, 
the documentation of the employment situation of participants attending a VTP is both a 
common practice and an uncontested criterion for evaluating the outcome of VTPs. For 
labour-market policy research the employment rate is the dominant injJicator of quality. 
This indicator is so dominant that within the framework of labour-market policy research 
the learning achievements of the trainees are sometimes ignored although this is 
another clearly product-orientated quality indicator for VTPs. Research into the training 
process is largely excluded from labour-market policy ex-post evaluations. 

With regard to the CIPP model, the dynamic relationship between process and product 
has yet to. be adequately researched. In product-orientated labour-market policy 
evaluations it is not only the situational conditions of the training process which have 
continuously been ignored in terms of their effects on the employment situation. The 
VTP participants' own subjective assessments, which are becoming increasingly 
significant in the training process research and quality management and which have 
already proven their worth, are often not taken into consideration when evaluating the 
employment integration quota. The same applies to the insights gained through the 
learning transfer research, which are given far too little attention both in the field of 
training process research and in the labour-market policy debate. 



Process-orientated evaluations are focused on the internal quality dimension of 
vocational training, which can be analysed using different criteria. These criteria could 
be a set of pre-formulated standards (like the ISO standards), or the demands of the 
target groups as well as the demands of the enterprises. The standards define the 
procedures to be followed and this approach implicitly supposes that following the 
procedures will guarantee the quality of the programme. Following the logic of 
evaluation approaches, these quality elements naturally contribute to achieving the 
objectives and, by extension, the products of VTPs. Whatever criteria are used, however, 
an evaluation of vocational training programmes which is not interested in the 
placement rates of the trainees will not be satisfactory. Otherwise, a programme could 
be qualified as excellent even if only one single trainee managed to find a job on 
completion of that course. 

Hence, information on the whereabouts of former VTP participants must increasingly be 
registered including qualitative employment or unemployment criteria and the personal 
variables of the participants. The advantage of the employment rate as an indicator of 
quality is that it can be measured and it is general enough for it to be used to measure 
very different training policies. However, sometimes the placement rate is attributed 
mechanically to the training programme, although there might by no causal link between 
the successful completion of training and the later job. It is the specific weakness of 
product-orientated labour-market research that its main indicator for quality, the 
placement rate, does not permit a distinction as to which variables have influenced 
access to employment by which degree. 
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A high placement rate is not in itself an indicator of quality; nor is a low placement rate 
the indicator of low standards. The well-known statement by Tanguy (1986) which 
considers the relationship between vocational training and employment to be 
‘introuvable’ (cannot be found) is all too superficial. In individual cases the relationship 
between training and employment is not necessarily a causal one but this does not 
change anything about the general plausibility of the connection. 



So it is indeed possible and common practice in evaluation research to compare the 
employment indicator with the results of different training courses and thus to build up a 
hierarchy between more and less effective measures. In the evaluation studies reviewed 
comparisons were also made of the employment rates of various target groups with the 
general tendencies on the labour-market. Within the framework of these comparative 
analyses, it was possible to identify the disadvantages facing women on entering 
employment on completion ofVTPs (cf. Seyfried, Buhler, 1995). Furthermore, placement 
rates are used as indicators of necessary adjustments to policies and the design of VET 
programmes taking into account the developments in the labour-markets. 

Nevertheless, given the weaknesses in respect of the causality problem, it is not at all 
satisfying from a scientific point of view to use the placement rate as the single indicator 
for the effectiveness or for the classification of different training measures as more or 
less successful. In scientific research different approaches have been developed to 
escape from the causality trap, one of which is the establishment of a control group 
design. Another possibility is the creation of a comparison group through a statistical 
regression analysis. 

Although the findings of the reviewed studies stress the importance of process- 
orientated evaluations, the existence of quality criteria in the training process cannot be 
seen as an expression of quality in itself but only because these quality aspects are 
supposed to improve the objectives of the training programme concerned. Therefore, 
process-orientated and product-orientated evaluations should not be seen as 
alternatives but as additional approaches. A coherent approach to VTP evaluation must 
overcome the gaps found between ‘process’ and ‘product’. For the future there needs to 
be greater convergence between the debates on training policy and labour-market 
policy, as well as greater efforts on all sides into coordinating coherent research 
approaches with consideration being given to process-orientated and product- 
orientated indicators of quality. 
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Evaluation of quality aspects in vocational training 
programmes 

Synthesis report 

The on-going worldwide technological and economic changes as well as persisting 
unemployment are at the origin of the multiplication of vocational training programmes 
in recent years. This extraordinary expansion of the VET sector seems to reach its 
limits though, due to restricted public funding in the EU Member States. Therefore 
questions arise as to which effects and under which conditions certain programmes 
produce. 

In other words, programmes are evaluated on their effectiveness. The methods of eval- 
uation as used in various countries become in their turn object of investigation, in an 
attempt to improve them. 

We hope that this publication contributes to this effort and to the development of a 
common culture of vocational training programme evaluations through learning from 
one another. 

Prof. Dr Erwin Seyfried 
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